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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH (S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- tf the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- tf NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)^ Responsive to communication(s) filed on 14 January 2004 . 
2a)S This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) 03 Claim(s) 1-41 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) S Claim(s) 7-47 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) Q The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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Final Rejection 
Response to Amendment 

1 . It is hereby acknowledged that the following papers have been received 
and placed on record in the file: Amendment A as received on 1-14-04. 

2. Claims 1-41, have been examined. 
Status of claims: 

3. Claims 1 -7, 9, 1 1 , 1 2, 1 4-1 8, 20-27, and 30-39 are rejected under 35 
U.S.C. 102(e) as being anticipated by Hirai et al., Patent #6,526,215, hereinafter 
Hirai. 

4. Claims 28 and 29 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Ratakonda, Patent # 5,956,026. 

5. Claims 8, 10, 13, 19, 40, and 41 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Hirai and Ratakonda. 

Claim Rejections - 35 USC §112 

6. Claims 1, 3, 5, 6, 10, 12, 13, 15, 17, 18, 24, and 28-41 are rejected under 
35 U.S.C. 112, first paragraph, because the specification, while being enabling 
for selecting one of a key frame or a key region, does not reasonably provide 
enablement for selecting one of a key frame and a key region. The specification 
does not enable any person skilled in the art to which it pertains, or with which it 
is most nearly connected, to make and use the invention commensurate in scope 
with these claims. The specification does not provide support for the creation of 
a synthetic key frame from both key frame and key region elements. 
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Claim Rejections - 35 USC § 102 

7. The following is a quotation of the appropriate paragraphs of 35 

U.S.C. 102 that form the basis for the rejections under this section made in this 

Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1 ) an application for patent, published under section 
122(b), by another filed in the United States before the invention by the applicant for patent or 
(2) a patent granted on an application for patent by another filed in the United States before 
the invention by the applicant for patent, except that an international application filed under 
the treaty defined in section 351(a) shall have the effects for purposes of this subsection of an 
application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

8. Claims 1-7, 9, 11, 12, 14-18, 20-27, and 30-39 are rejected under 35 
U.S.C. 102(e) as being anticipated by Hirai et al., Patent # 6,526,215, hereinafter 
Hirai. With regard to claim 1 , which teaches a method of generating key frames 
comprising the steps of: receiving a video stream and dividing it into meaningful 
sections, selecting the key frames, and combining key frames to form one key 
frame, Hirai teaches, in column 4, line 45-64 and column 10, lines 40-45, an 
apparatus for receiving moving picture data, dividing it into scenes, further 
dividing it up into cuts, and forming still images as representatives (keyframes) 
of these subdivisions. 

9. With regard to claim 2, which teaches the dividing step including receiving 
video form a second source, Hirai teaches, in column 3, line 8 and in figure 5, 
receiving input from sources such as a movie, and a still image. 

10. With regard to claim 3, which teaches the selecting step of including key 
frames output from a second source, Hirai teaches, in column 3, line 8 and in 
figure 5, receiving input from sources such as a movie, and a still image. 
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1 1 . With regard to claim 4, which teaches that a section is a unit of a segment, 
Hirai teaches, in column 4 line 45, that a scene (section) comprises a plurality of 
cuts (segments). 

12. With regard to claim 5, which teaches a method of describing a key frame 
comprising, dividing a video into meaningful segments, and describing a list of 
keyframes, Hirai teaches, in column 4, lines 50-52, a detecting means for 
designating said representative still images of said scenes and cuts to edit said 
moving picture. Hirai also teaches, in column 15, line 26, that video data 
belonging to the M-icon (key frame) is displayed in the form of a list. 

13. With regard to claim 6, which teaches the describing step including: an ID, 
a representative segment locator, a key frame list for identifying elements of a 
keyframe, and also include a fidelity value, and information on arrangement, 
Hirai teaches, in column 3, line 8, and in figures 4 and 5, video linked information 
including names, types, and lengths of media. Hirai further teaches, in column 
15, line 26, video data belonging to the M-icon that is displayed in the form of a 
list. Hirai further teaches, with regard to the items that be included, in column 5, 
line 6, a degree of abstraction, and in column 9, lines 35-51 in conjunction with 
figures 8 and 10, Hirai teaches the arrangement of parent and child elements in a 
Hierarchy. 

14. With regard to claim 7, which teaches the information about the 
arrangement includes 2-D or 3-D location information, Hirai teaches, in column 
19, line 50, that there is preparation of items in a hierarchical structure so that 
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when information is not suitable, one of the items in a layer higher that the layer 
containing the information item can be used. 

15. With regard to claim 9, which teaches a key frame locator including: the 
location, an annotation, and a related segment, also a segment locator 
designates a segment represented by a corresponding key frame, and a fidelity 
value, Hirai teaches, in column 11, line 9-15, information regarding address, 
name, and cut start/end points. Hirai also teaches, in column 5, line 6, that the 
degree of abstraction is determined for the images. 

16. With regard to claim 11, which teaches an s-key frame including a key 
frame list, Ratakonda teaches, in column 15, line 26, that M-icons (key frames) 
can contain a list of M-icons. With regard to claim 1 1 also teaching each element 
of a key frame list having a fidelity value, Ratakonda teaches in column 5, line 6, 
that the degree of abstraction is determined for the images. 

17. With regard to claim 12, which teaches a method of describing key frame 
data comprising: dividing a video into meaningful sections, and generating the 
key frames. Hirai teaches, in column 4, line 45-64 and column 10, lines 40-45, 
an apparatus for receiving moving picture data, dividing it into scenes, further 
dividing it up into cuts, and forming still images as representatives (keyframes) 
of these subdivisions. 

18. With regard to claim 14, which teaches that each element of a key frame 
list contains a keyframe locator, Hirai teaches, in column 11, lines 9-15, that 
layer and address information for each M-icon (keyframe). 
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19. With regard to claim 15, which teaches a key region locator including an 
ID, and a representative segment locator, that includes: a fidelity value, a 
annotation, and a list of related segments, Hirai teaches, in column 3, line 38 and 
in figures 4 and 5, source and individual frame names. Hirai also teaches a 
means of calculating a fidelity value (see column 5, line 6), an annotation (see 
column 11, lines 9-15), and a list of related segments (see column 11, lines 9- 
15). 

20. With regard to claim 1 6, which teaches each element of a key region list 
including a fidelity value, Hirai teaches, in column 5, line 6, that the images 
contain a degree of abstraction. 

21 . With regard to claim 17, which teaches information about arrangement 
includes either a 2-D or a 3-D locator, Hirai teaches, in column 19, line 50, the 
tree structured arrangement of the information has an advantage that the tag 
buttons can be easily searched. 

22. With regard to claim 18, which teaches a hierarchical video summary 
method comprising means of, dividing a video stream into meaningful sections, 
assigning key frames to a key image locator, a hierarchical summary list for 
describing lower summary structures, and structural information, Hirai teaches, in 
column 4, line 45-64 and column 10, lines 40-45, receiving moving picture data, 
dividing it into scenes, further dividing it up into cuts, and forming still images as 
representatives (key frames) of these subdivisions. Hirai further teaches, in 
column 11, lines 9-15, information being contained in the elements of a 
hierarchical structure giving address information and video structural information. 
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23. With regard to claim 20, which teaches that each hierarchical summary 
structure is represented by an image representative of a specific segment, Hirai 
teaches, in column 16, line 1 and in figure 1, how in the hierarchy each M-icon 
(key frame) has its own information zone. 

24. With regard to claim 21 , which teaches that each component of the lower 
hierarchical summary list uses a hierarchical/recursive summary structure as a 
lower hierarchical summary structure, Hirai teaches, in column 9, line 7 and in 
figure 1 , how the hierarchy is organized from the top level story to the next level 
scenes to the next level of cuts where a cut is a subset of a scene, and each of 
these M-icons (keyframes) has a summary element. 

25. With regard to claim 22, which teaches that the hierarchical summary 
structure has summary level information, Hirai teaches, in column 10, lines 56- 
67, that each icon is given a layer level value (1,2,... from the bottom layer). 

26. With regard to claim 23, which teaches the hierarchical summary structure 
including a fidelity value, Hirai teaches, in column 5, line 6, and a level of degrees 
of abstraction as associated with frame images. 

27. With regard to claim 24, which teaches a method for providing a video 
browsing interface comprising: dividing a video stream into meaningful sections, 
and providing a user interface to a predetermined display, Hirai teaches, in 
column 4, line 45-64 and column 10, lines 40-45, an apparatus for taking in 
moving picture data and dividing it up into scenes and cuts. Hirai also teaches in 
column 4, line 60, a means for displaying said hierarchical structure. 
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28. With regard to claim 25, which teaches the user interface providing the 
synthetic key frame in the form of view, Hirai teaches in figure 1 , a visual 
representation of the M-icons (key frames). 

29. With regard to claim 26, which teaches key frames being arranged in a 
time sequence, and the key frames arranged in a tree shape, Hirai teaches, in 
column 9, lines 35-51 and in conjunction with figures 8 and 10, how all of the M- 
icons are arranged in the order in which they occurred in the inputted moving 
picture, and that they are displayed in a hierarchical tree structure. 

30. With regard to claim 27, which teaches key frames assigned to each node 
in a TOC form, Hirai teaches, in column 9, lines 25-34 and in conjunction with 
figures 8 and 10, managing information in an organized form, and then displays a 
TOC (figure for the hierarchy in figure 10. 

31 . With regard to claims 30, 32, 34, 36, and 38, which teach the synthetic key 
frame including one of a selected key frame and a selected key region from each 
of the plurality of sections, Hirai teaches, in column 4, lines 45-64, a group of 
frames combined into a representative frame. 

32. With regard to claims 31 , 33, 35, 37, and 39, which teach each of the 
plurality of sections comprising a video frame, and the selected one of the key 
frame and the key region comprises a portion of the video frame, Hirai teaches, 
in column 4, lines 45-64, a group of frames (video sequence or clip) combined 
into a representative frame and in column 3, lines 1-4, the association between 
the pictures in the hierarchal structure and their associated moving picture 
sequences. 
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33. Claims 28 and 29 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Ratakonda, Patent # 5,956,026. With regard to claim 28, which 
teaches dividing video into meaningful segments, Ratakonda teaches in column 
2, lines 13-27 and in column 9, lines 40-43, generating a summary of a video 
based on key frames by detecting shot boundaries, and locating shots. With 
regard to claim 28 also teaching providing a user interface, Ratakonda teaches, 
in column 13, line 35, providing a user interface. With regard to claim 28 
teaching selecting s-key frames according to the user input, and reproducing a 
segment represented by the selected s-key frame, Ratakonda teaches, in column 
5, lines 56-63, clicking on a particular frame and being able to display a normal 
playback of a video sequence. 

34. With regard to claim 29, which teaches a reproducing step that reproduces 
a segment related with constituent elements of the contents of the key frame, 
Ratakonda teaches in column 5, lines 56-63, clicking on a particular frame and 
being able to display a normal playback of a video sequence. 

Claim Rejections - 35 USC § 103 

35. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 

all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 1 02 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

36. Claims 8, 10, 13, 19,40, and 41 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Hirai and Ratakonda. Hirai teaches a method of 
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generating representative still images from a video sequence, a locating means 
comprising: a means of calculating a fidelity value (see column 5, line 6), a 
annotation (see column 1 1 , lines 9-15), a list of related segments (see column 
11, lines 9-15), and information on arrangement (see column 11, lines 9-15), but 
doesn't teach elements in a key frame list containing locators, the locator 
including an inherent ID, and image locator to locate the image, a representative 
segment locator, or a key image locator being a structure for designating an 
image using, a key image locator, a key frame locator, a s-key frame locator. 
Ratakonda teaches a video summarization system similar to that of Hirai, but 
further teaches elements in a key frame list containing locators, the locator 
including an inherent ID, and image locator to locate the image, a representative 
segment locator, and a key image locator being a structure for designating an 
image using a, key image locator, a key frame locator, a s-key frame locator. 
With regard to claim 8, Ratakonda teaches, in column 6, lines 45-67, a means for 
locating items in a key frame list. It would have been obvious to one of ordinary 
skill in the art, having the teachings of Hirai and Ratakonda before him at the 
time the invention was made to modify the method of generating representative 
still images of a video sequence of Hirai to include the representative image 
locating means of Ratakonda. One would have been motivated to make such a 
combination because locating these key frames and the video segment they 
represent can help in editing and/or viewing the video sequence. 
37. With regard to claim 10, which teaches the locator including an inherent 
ID, an image locator to locate the region, a representative segment locator, a key 
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region locator that can contain: a fidelity value, an annotation, and a list of 
related segments (all rejected supra). Ratakonda teaches, in column 6, line 60, a 
key frames having specific IDs. Ratakonda further teaches, in column 6, line 63, 
the region that a key frame represents. Ratakonda also teaches, in column 6, 
line 62, that the key frame kj represents a contiguous set of video frames. It 
would have been obvious to one of ordinary skill in the art, having the teachings 
of Hirai and Ratakonda before him at the time the invention was made to modify 
the method of generating still images of Hirai to include the key frames having 
specific Ids, the region that a key frame represents, and the key frame kj 
representing a contiguous set of video frames of Ratakonda. One would have 
been motivated to make such a combination because the more information that 
is available, the easier it will be for the user to use the video software to 
effectively find the sought after segment, find out how representative it is, and 
find relative segments. 

38. With regard to claim 13, which teaches the locator including an inherent 
ID, and image locator to locate the region, a representative segment locator, a 
key region list, a fidelity value (as rejected supra), and information on 
arrangement (as rejected supra), Ratakonda teaches, in column 6, line 60, key 
frames having specific IDs. Ratakonda further teaches, in column 6, line 63, the 
region that a keyframe represents. Ratakonda also teaches, in column 6, line 
62, that the key frame kj represents a contiguous set of video frames. Ratakonda 
further teaches, in column 6, line 62, a list of the frames that a key frame 
represents. It would have been obvious to one of ordinary skill in the art, having 
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the teachings of Hirai and Ratakonda before him at the time the invention was 
made to modify the method of generating still images of Hirai to include the key 
frames having specific Ids, the region that a key frame represents, that the key 
frame kj represents a contiguous set of video frames, and a list of the frames that 
a key frame represents. One would have been motivated to make such a 
combination because the more information that is available, the easier it will be 
for the user to use the video software to effectively find the sought after segment, 
find out how representative it is, and find relative segments. 

39. With regard to claim 1 9, which teaches a key image locator being a 
structure for designating an image using: a key image locator, a key frame 
locator, and a s-key frame locator, Ratakonda teaches, in column 6, lines 45-67, 
a locator method for locating key images of all levels of the hierarchy, and all the 
frames which they represent. It would have been obvious to one of ordinary skill 
in the art, having the teachings of Hirai and Ratakonda before him at the time the 
invention was made to modify the method of generating still images of Hirai to 
include a locator method for locating key images of all levels of the hierarchy, 
and all the frames which they represent of Ratakonda. One would have been 
motivated to make such a combination because the inclusion of these location 
and identification elements will help the user in using the video software. 

40. With regard to claim 40, which teaches the synthetic key frame including 
one of a selected key frame and a selected key region from each of the plurality 
of sections, Hirai further teaches, in column 4, lines 45-64, a group of frames 
combined into a representative frame. 
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41 . With regard to claim 41 , which teaches each of the plurality of sections 
comprising a video frame, and the selected one of the key frame and the key 
region comprises a portion of the video frame, Hirai further teaches, in column 4, 
lines 45-64, a group of frames combined into a representative frame and in 
column 3, lines 1-4, the association between the pictures in the hierarchal 
structure and their associated moving picture sequences. 

Response to Arguments 

42. The arguments filed on 1 1-13-2003 have been fully considered but they 
are not persuasive. The reasons are set forth below. 

43. With respect to the applicants argument, that Hirai does not teach or 
suggest selecting one of a key frame and a key region from each of a plurality of 
selections and combining the selected one of the key frames and the key region 
from each of the plurality of scenes to form a synthetic key frame, as recited in 
independent claim 1 . 

44. In response, the examiner respectfully submits that Hirari does teach, in 
column 4, lines 45-64 and in column 5, lines 15-26, dividing a video sequence up 
in to scenes, cuts, and still images, and displaying them in an hierarchical format 
with each having a respective still images representing a section of the original 
video sequence. Hirari supports this idea in column 10, lines 40-45, where he 
mentions combining still images to form an new still image for the hierarchical 
structure. 
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45. With respect to the applicants argument, that Ratakonda does not teach or 
suggest synthesizing key frames or key regions into one image so as to generate 
a synthetic key frame. 

46. In response, the examiner respectfully submits, that Ratakonda does 
teach, in column 9, lines 40-43, clustering keyframes to form representative 
keyframes. 



Conclusion 

47. Applicant's amendment necessitated the new ground(s) of rejection 
presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. 
See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as 
set forth in 37 CFR 1.136(a). 

48. A shortened statutory period for reply to this final action is set to expire 
THREE MONTHS from the mailing date of this action. In the event a first reply is 
filed within TWO MONTHS of the mailing date of this final action and the advisory 
action is not mailed until after the end of the THREE-MONTH shortened statutory 
period, then the shortened statutory period will expire on the date the advisory 
action is mailed, and any extension fee pursuant to 37 CFR 1 .136(a) will be 
calculated from the mailing date of the advisory action. In no event, however, will 
the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

49. Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Dennis G Bonshock whose telephone number 
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is (703)305-4668. The examiner can normally be reached on Monday - Friday, 
8:30 a.m. - 5:00 p.m.. 

50. If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, John Cabeca can be reached on (703)308-31 16. The fax 
phone number for the organization where this application or proceeding is 
assigned is 703-872-9306. 

51 . Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status information 
for published applications may be obtained from either Private PAIR or Public 
PAIR. Status information for unpublished applications is available through 
Private PAIR only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll- 
free). / 
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